The present invention generally relates to image registration and, more particularly, to image registration using a two stage registration process.
Two different images of a common scene are often obtained so that different categories of information within the scene can be captured. For instance, the images may represent reflectances in different spectra, such as a first image in the visible wavelengths and the other image in the infrared wavelengths. One image may be a synthetic image, such as from synthetic aperture radar, and the other image may be an electro-optical image, such as from a visible or infrared system. On the other hand, both images may be synthetic, such as from a synthetic aperture radar and a digital graphic map.
To understand the images captured, image registration is required. Image registration involves the computation of a mathematical transformation which maps or transforms pixel coordinates of one image into the associated pixel coordinates of the other image. Digital images often need to be registered so that pixels associated with specific locations in one image can be associated with pixels of exactly the same locations in a second image.
Image registration can be quite difficult because of the number of degrees of freedom involved in the registration process. The parameters involved include translation in two dimensions, rotation around a point in the image plane, and perspective of the scene from the viewpoint (which involves the relative displacement between the observation point and the central point of view, the orientation of the imaging device at that point, and the imaging characteristics of the imaging device, such as focal length, etc.).
In order to reduce image registration difficulty, registration is typically performed between images that have the same viewing geometry of the same scene. In this way, the images have large differences in only two parameters, that is, translation in the coordinate axes of the images, e.g., X and Y. Although other parameters may vary, those differences constrain the applicability and usability of the registration process by introducing error that can defeat or mislead the registration operation.
Numerous techniques, primarily based on statistical correlation or information measures, have been developed and improved to solve offset differences with optimal performance. Such techniques affect a rapid and efficient search through the range of possible offsets and generate a xe2x80x9cgoodness of fitxe2x80x9d measure at each hypothesized offset. The location offering the best fit is taken to derive the offset mapping.
On the other hand, relative rotation between images is extremely expensive to compute. Techniques comparable to correlation or information measures have not yet been developed to rapidly and efficiently search through the space of possible rotations. In addition, rotational registration couples with the offset since rotation of one image is seen as being around some point in its plane. As a result, searching is typically done using a brute force technique. For example, one image is rotated, then a rapid offset registration process is performed, and the sequence is repeated until a best goodness of fit measure is obtained. The rotation at which that best fit is obtained is then taken to indicate the relative rotation between the images.
Although many approaches have been employed to improve rotational registration, they all suffer from certain deficiencies. Such deficiencies include multiple reduced resolution stages which require extra processing, rotational matching impracticality due to computational requirements, unsuitability due to expensive Fourier methods, and lack of quality assurance to evaluate the registration mapping. In view of the foregoing, an improved registration process is needed.
In view of the foregoing, it is in object of the present invention to provide a two-stage process with a reduced resolution stage first serving to give approximate parameters that are refined in a subsequent stage in a computationally efficient manner.
It is another object of the present invention to use a reduced resolution stage so that brute force matching over a range of rotational differences can be effectively applied.
It is yet another object of the present invention to obtain both angle and offset approximations during the reduced resolution stage to make the subsequent stage more efficient.
It is still yet another object of the present invention to register images of vastly different resolutions.
It is another object of the present invention to use small patch registration in a partially registered image to make control point matching more efficient.
It is yet another object of the present invention to use patch location adjustment to make control point matching more robust.
It is still yet another object of the present invention to use patch layout and patch matching evaluation to measure control point match quality.
It is another object of the present invention to use geometric figure mapping from a fiducial image space to the registered image space to determine registration quality.
With the foregoing objects in mind, the present invention includes capturing a first original image of a scene under surveillance, capturing a second original image of the scene under surveillance, coarsely matching the first original image to the second original image to give approximate angle and offset parameters of the first original image to the second original image, and finely matching the first original image to the second original image based on the approximate angle and offset parameters of the coarsely matching step. The coarsely matching step involves applying a plurality of rotations to a reduced version of the first original image to form a rotated and reduced first original image and performing a correlation match between the rotated and reduced first original image and a reduced version of the second original image. The finely matching step employs control point matching between the first original and the second original image using small image patches at the scale of the lower resolution of the two images.
During coarse matching, the reduced first original image is rotated by a sequence of angle increments that in combination encompass a range of possible rotational mismatches between the first original image and the second original image. The rotated and reduced first original image and the reduced second original image are transformed into the Fourier domain by subjecting each to a Fourier transform to produce Fourier images including an array of spatial frequency coefficients. A complex conjugate transform is then applied to one of the Fourier images. Thereafter, the Fourier images are correlated by pixel multiplication in the Fourier domain to yield a product image.
An inverse Fourier transform is applied to the product image to yield a correlation image having pixel values representing correlation coefficients corresponding to offsets given by row and column coordinates of a correlation coefficient pixel. A peak correlation coefficient is determined for the correlation image by examining all the correlation coefficients and identifying the largest one, and its location by row and column offset in the correlation image. A set of peak correlation coefficients is collected, one for each different rotation of the rotated and reduced first original image, along with a correlation peak-to-side-lobe (PSR) value, correlation histogram, and the peak location. A rotational difference is then identified between the first and second original images by examining the set of peak correlation coefficients for a maximum correlation coefficient. The PSR and correlation histogram are evaluated to determine if the maximum peak is of sufficient quality to proceed. If it is not, a secondary peak having the best PSR and correlation histogram is selected as the maximum peak. An approximate shift offset is identified between the first and second original images by determining row and column offsets of the maximum peak correlation coefficient corresponding to the rotational difference.
To prepare the images for fine matching, the first original image is reduced to form a second reduced first original image having a common scenic scale as the second original image so that scenic objects appearing in both images have a common extent. The second reduced first original image is then rotated by the rotational difference to form a second rotated and reduced first original image.
The finely matching step involves selecting a plurality of first offset patch images from the second rotated and reduced first original image from a grid of pre-selected locations uniformly and symmetrically spread around an extent of the second rotated and reduced first original image as modified by shifting according to the row and column offsets. Each first offset image patch is then checked to ensure that each first offset image patch falls completely within the extent of the second rotated and reduced first original image. Next, a plurality of second image patches are selected from a grid of pre-selected locations uniformly and symmetrically spread around an extent of the second original image.
The first offset image patches are then matched with the second offset image patches by correlation such that each pair of corresponding patches produces one correlation image that is used to produce one control point for image registration. The peak correlation value in each correlation image is identified during correlation. The location of the peak correlation value gives the precise offset between the match point in the second original image and a corresponding match point in the second reduced and rotated first original image.
The precise offset determined during the correlation step and the row and column offset determined in the coarsely matching step are then added to the location of the second offset image patch from the second original image to produce a measured location equivalent to a point in the second reduced and rotated first original image that corresponds with the match point in the second original image.
Next, the measured location is de-rotated by applying the reverse of the rotation that was applied to the second reduced first original image and re-scaled by applying the reverse of the second reduction that was applied to the first original image to yield a correct control point location in the first original image that matches a corresponding control point in the second original image. The set of all control point pairs gives a control set which is used to obtain a mathematical transformation that provides registration mapping between the first and second original images. The control set is a set of pairs of corresponding measured and fiducial points in the first and second original images that define which points in the images coincide.
Registration mapping involves a pair of mathematical functions that map coordinate locations in the first original image into the corresponding locations in the second original image. A statistical adjustment is used to produce a set of parameters that specify the mathematical functions.
According to another aspect of the present invention, a measure of remaining distance errors is determined between control point pairs after registration mapping to indicate an accuracy of the registration mapping of the control point pairs in the control set. The errors are then inspected to determine if the registration mapping produces excessive error. Preferably, the inspecting step includes computing a sequence of point coordinates corresponding to a geometric shape in the second original image, transforming the sequence of point coordinates using the registration mapping into point coordinates in the first original image, and evaluating the transformed points to determine a closeness of fit of the sequence of point coordinates relative to the geometric shape in the second original image.